A Robust Bayesian Truth Serum for Small Populations

Peer prediction mechanisms allow the truthful elicitation of private signals (e.g., experiences, or opinions) in regard to a true world state when this ground truth is unobservable. The original peer prediction method is incentive compatible for any number of agents n ≥ 2, but relies on a common prior, shared by all agents and the mechanism. The Bayesian Truth Serum (BTS) relaxes this assumption. While BTS still assumes that agents share a common prior, this prior need not be known to the mechanism. However, BTS is only incentive compatible for a large enough number of agents, and the particular number of agents required is uncertain because it depends on this private prior. In this paper, we present a robust BTS for the elicitation of binary information which is incentive compatible for every n ≥ 3, taking advantage of a particularity of the quadratic scoring rule. The robust BTS is the first peer prediction mechanism to provide strict incentive compatibility for every n ≥ 3 without relying on knowledge of the common prior. Moreover, and in contrast to the original BTS, our mechanism is numerically robust and ex post individually rational. Introduction Web services that are built around user-generated content are ubiquitous. Examples include reputation systems, where users leave feedback about the quality of products or services, and crowdsourcing platforms, where users (workers) are paid small rewards to do human computation tasks, such as annotating an image. Whereas statistical estimation techniques (Raykar et al. 2010) can be used to resolve noisy inputs, for example in order to determine the image tags most likely to be correct, they are appropriate only when user inputs are informative in the first place. But what if providing accurate information is costly for users, or if users otherwise have an external incentive for submitting false inputs? The peer prediction method (Miller, Resnick, and Zeckhauser 2005) addresses the quality control problem by providing payments (in cash, points or otherwise) that align an agent’s own interest with providing inputs that are predictive of the inputs that will be provided by other agents. Formally, the peer prediction method provides strict incentives for providing truthful inputs (e.g., in regard to a user’s information about the quality of a product, or a user’s view on Copyright c © 2012, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. the correct label for a training example) for a system of two or more agents, and when there is a common prior amongst agents and, critically, known to the mechanism. The Bayesian Truth Serum (BTS) by Prelec (2004) still assumes that agents share a common prior, but does not require this to be known by the mechanism. In addition to an information report from an agent, BTS asks each agent for a prediction report, that reflects the agent’s belief about the distribution of information reports in the population. An agent’s payment depends on both reports, with an information component that rewards reports that are “surprisingly common,” i.e., more common than collectively predicted, and a prediction component that rewards accurate predictions of the reports made by others. A significant drawback of BTS is that it only aligns incentives for a large enough number of agents, where this number depends on the prior and is thus unknown to the mechanism. In addition, BTS may leave a participant with a negative payment, and is not numerically robust for all inputs. In this paper, we present the robust Bayesian Truth Serum (RBTS) mechanism, which, to the best of our knowledge, is the first peer prediction mechanism that does not rely on knowledge of the common prior to provide strict incentive compatibility for every number of agents n ≥ 3. RBTS is also ex post individually rational (so that no agent makes a negative payment in any outcome) and numerically robust, being well defined for all possible agent reports. Moreover, the mechanism seems conceptually simpler than BTS, and the incentive analysis is more straightforward. The main limitation of RBTS relative to earlier mechanisms, is that it applies only to the elicitation of binary information; e.g., good or bad experiences, or true or false classification labels.1 Extending RBTS to incorporate more than two signals is the most important direction for future research. RBTS takes the same reports as BTS, and an agent’s payment continues to consist of one component that depends on an agent’s information report and a second component that Many interesting applications involve binary information reports. This is supported by the fact that Prelec’s own experimental papers have adopted the binary signal case (Prelec and Seung 2006; John, Loewenstein, and Prelec 2011). Indeed, as the number of possible information reports increases, so does the difficulty imposed on users in providing the prediction report, which must include estimates for the additional possible information reports. depends on an agent’s prediction report. The main innovation is to induce a “shadow” posterior belief report for an agent i from her information report and the prediction report of another agent j, adjusting this prediction report in the direction suggested by agent i’s information report. We couple this with a particularity of the quadratic scoring rule, by which an agent prefers a shadow posterior belief that is as close as possible to her true posterior. In order to determine the agent’s payment, we then apply both the shadow posterior belief and the agent’s prediction report to the quadratic scoring rule, adopting the information report of a third agent k as the event to be predicted.